Beyond the GPU: Engineering an AI-Ready Infrastructure for Distributed Training at Scale
The rapid evolution of artificial intelligence workloads is fundamentally altering the requirements for modern cloud-native infrastructure, necessitating a shift from simple cluster provisioning to the creation of highly specialized, performance-oriented environments. As Atlassian’s ML platform engineers Abhi Kulkarni and Shishir Jindal recently detailed, the traditional "AI-ready" designation—which historically meant little more than deploying a Kubernetes cluster with GPU access—is no longer sufficient. When training models that span multiple nodes, the primary technical bottlenecks shift from raw compute power to critical infrastructure layers: inter-node communication, shared storage, topology awareness, and automated system validation.
The challenge at Atlassian emerged as their internal machine learning platform, which supports product experiences like search and ranking, began to face scalability constraints. As models transitioned into the "tens of billions of parameters" category, single-node training became physically impossible. The combined requirements of model weights, optimizer states, and workable batch sizes exceeded the memory capacity of any individual machine, forcing a transition to distributed training architectures. This shift necessitated a platform that could guarantee reliability and repeatability without overwhelming ML researchers with the underlying complexities of cloud networking and storage fabrics.
A Chronology of Infrastructure Evolution
For several years, the approach to distributed training was largely reactive. Before the integration of high-performance technologies like Remote Direct Memory Access (RDMA) and Lustre, distributed jobs were executed on a "best-effort" basis. Platform teams often lacked visibility into why specific jobs performed poorly, treating latency issues as isolated anomalies rather than systemic design flaws.
The turning point for Atlassian came through the identification of "silent failures." In one instance, a misconfiguration caused high-bandwidth collective traffic to default to standard network sockets rather than the intended RDMA fabric. Because the job still completed—albeit at a fraction of the expected speed—the degradation went unnoticed for months. In another critical incident, a device plugin responsible for advertising RDMA fabric to Kubernetes remained in a CrashLoopBackOff state for 271 days. The failure was only discovered by chance during an unrelated GPU operator upgrade. The root cause was a mundane mismatch in container image manifest architectures, highlighting a significant blind spot in production monitoring: the system lacked synthetic validation, meaning it only monitored what was actively running, not the state of the idle, unused high-performance fabric.
The Shift to Integrated Platform Design
Recognizing that tuning individual components was insufficient, the engineering team pivoted toward a holistic platform design. They stopped viewing RDMA networking, shared storage, and topology mapping as separate technical hurdles and began treating them as a single, unified constraint.
This shift in strategy required systemic changes across the entire infrastructure stack. By integrating RDMA-capable networking, the platform moved away from hidden bottlenecks inherent in standard TCP paths. Similarly, replacing inefficient storage access patterns with Lustre-based shared, high-throughput storage allowed for concurrent checkpointing and dataset access. The final, and arguably most critical, component was the introduction of proactive validation. By moving away from purely reactive dashboards—which only reflect successful runs—to a system that actively verifies the transport and storage path before and during job execution, the platform team effectively eliminated the risk of silent performance degradation.
Data-Driven Performance Gains
The move to an integrated infrastructure yielded measurable performance improvements that confirm the necessity of treating transport and storage as first-class citizens. In a series of controlled benchmarks, the contrast between the old "socket-based" environment and the new RDMA-capable architecture was stark.
When running Qwen2.5-14B FSDP supervised fine-tuning across 16 H200 GPUs distributed over two nodes, the median step time dropped from 12.36 seconds to 6.07 seconds. Perhaps more impressive was the training throughput improvement, which saw a 2.04x increase in efficiency under identical model-load phases. Furthermore, the peak bus bandwidth reached 355 GB/s using 2-node NCCL all-reduce operations. These metrics demonstrate that in distributed AI, performance is not merely an optimization goal; it is a requirement for operational correctness.
Broader Implications for the Cloud-Native Ecosystem
The challenges faced by Atlassian are not unique to any single organization; they represent a growing pain point for any enterprise moving into large-scale generative AI. As adoption grows, platform teams are increasingly hitting the same wall: accelerators, network fabrics, and storage layers are currently treated as distinct, disjointed domains.
The industry is currently witnessing a transition where cloud-native orchestration—traditionally designed for microservices and stateless web applications—must adapt to the high-throughput, low-latency requirements of stateful, distributed AI training. The lesson for the broader community is clear: "the job ran" is no longer an acceptable success signal. Organizations must implement validation frameworks that confirm high-performance jobs are actually utilizing the intended hardware paths.
From a management perspective, this implies that platform teams must absorb the complexity of physical topology. Atlassian’s findings suggest that RDMA, for instance, is inextricably linked to the physical location of hardware within a datacenter. Reservations that move across zones can break network mappings, leading to performance cliffs. By automating the transition of node pools and treating infrastructure changes as controlled platform migrations, teams can maintain stability even as they scale.
Conclusion and Strategic Outlook
As artificial intelligence models continue to increase in size and complexity, the demand for sophisticated, integrated infrastructure will only intensify. The core takeaway for engineers and architects is to move away from viewing networking, storage, and scheduling as a collection of independent features. Instead, these components must be synthesized into a single, cohesive product experience.
The success of such platforms depends on their ability to mask infrastructure complexity while providing the necessary guardrails to ensure that distributed training remains predictable and efficient. As demonstrated by the integration of tools like Kubeflow and Kubernetes within the Atlassian ecosystem, the future of AI infrastructure lies in building systems that treat performance-critical transport and storage paths with the same rigor as application code. By focusing on visibility, synthetic validation, and integrated design, organizations can bridge the gap between "running a job" and achieving industrial-scale AI production readiness. The era of treating infrastructure as a passive utility is over; the era of the integrated, performance-aware platform has begun.
